Evaluating Rank Accuracy based on Incomplete Pairwise Preferences
نویسندگان
چکیده
Existing methods to measure the rank accuracy of a recommender system assume the ground truth is either a set of user ratings or a total ordered list of items given by the user with possible ties. However, in many applications we are only able to obtain implicit user feedback, which does not provide such comprehensive information, but only gives a set of pairwise preferences among items. Generally such pairwise preferences are not complete, and thus may not deduce a total order of items. In this paper, we propose a novel method to evaluate rank accuracy, expected discounted rank correlation, which addresses the unique challenges of handling incomplete pairwise preferences in ground truth and also puts an emphasis on properly ranking items that users most prefer.
منابع مشابه
Partial Ranking by Incomplete Pairwise Comparisons Using Preference Subsets
In multi-criteria decision making the decision maker need to assign weights to criteria for evaluation of alternatives, but decision makers usually find it difficult to assign precise weights to several criteria. On the other hand, decision makers may readily provide a number of preferences regarding the relative importance between two disjoint subsets of criteria. We extend a procedure by L. V...
متن کاملStatistical Inference for Incomplete Ranking Data: The Case of Rank-Dependent Coarsening
We consider the problem of statistical inference for ranking data, specifically rank aggregation, under the assumption that samples are incomplete in the sense of not comprising all choice alternatives. In contrast to most existing methods, we explicitly model the process of turning a full ranking into an incomplete one, which we call the coarsening process. To this end, we propose the concept ...
متن کاملLearning from Pairwise Preference Data using Gaussian Mixture Model
In this paper we propose a fast online preference learning algorithm capable of utilizing incomplete preference information. It is based on a Gaussian mixture model that learns soft pairwise label preferences via minimization of the proposed soft rank loss measure. Standard supervised learning techniques, such as gradient descent or Expectation Maximization can be used to find the unknown model...
متن کاملData-driven Rank Breaking for Efficient Rank Aggregation
Rank aggregation systems collect ordinal preferences from individuals to produce a global ranking that represents the social preference. Rank-breaking is a common practice to reduce the computational complexity of learning the global ranking. The individual preferences are broken into pairwise comparisons and applied to efficient algorithms tailored for independent paired comparisons. However, ...
متن کاملData-driven rank ordering - a preference-based comparison study
Data driven rank ordering refers to the rank ordering of new data items based on the ordering inherent in existing data items. This is a challenging problem, which has received increasing attention in recent years in the machine learning community. Its applications include product recommendation, information retrieval, financial portfolio construction, and robotics. It is common to construct or...
متن کامل